Comparison of Modern Stochastic Optimization Algorithms

Gradient-based optimization methods are popular in machine learning applications. In large-scale problems, stochastic methods are preferred due to their good scaling properties. In this project, we compare the performance of four gradient-based methods; gradient descent, stochastic gradient descent, semi-stochastic gradient descent and stochastic average gradient. We consider logistic regression with synthetic data and softmax regression on the MNIST dataset of handwritten digits.